A “cluster” is the result of “clustering” computing resources together in such a way that they behave like a single resource. Clustering is often used for purposes of parallel processing, load balancing and fault tolerance. One common example of a cluster is a set of computers, or “nodes”, that are configured so that they behave like a single computer. Each computer in the cluster has shared access to a set of resources. A resource is, generally, any item that can be shared by the computers in the cluster. A common example of a resource is a block of memory in which information is stored. The block of memory may be part of a node in the cluster or may be external to the cluster, such as a database block.
A cluster comprises multiple nodes that each executes an instance of a server that each facilitates access to a shared set of resources on behalf of clients of the cluster. One example of a cluster is a database cluster. A database cluster comprises multiple nodes that each executes an instance of a database server that each facilitates access to a shared database. Among other functions of database management, a database server governs and facilitates access to the particular database by processing requests by clients to access data in the database.
Because the several nodes of a cluster share access to a database, multiple nodes or database server instances might attempt to access the same data resource (e.g., data block on disk) concurrently with each other. One of the nodes might obtain a lock on the sought data resource before the other nodes do, thereby forcing the other nodes to wait for the lock-holding node to finish its work relative to the data resource and release the lock.
Sometimes, a node needs to obtain a lock on multiple separate data resources before that node can complete a transaction. For example, a transaction might involve deducting an amount from one bank account and adding that amount to another bank account. Such work often needs to be done atomically, such that neither bank account will be affected under circumstances in which the change cannot be made to one or the other of the banks accounts. The data for the first bank account might be stored in a first data block, while the data for the second bank account might be stored in a second data block. Under such circumstances, a database server instance might need to obtain a lock on both the first data block and the second data block before modifying the data in either data block. If some other instance holds a lock on either of the data blocks, then the database server instance typically will be forced to wait for the other instance to release its lock on the locked data block.
Difficulties sometimes arise in cases where multiple nodes end up waiting for the release of locks that another other node holds. For example, node “A” might need to obtain locks on resources “1” and “2” in order to perform a task. Node “B” might need to obtain locks on resources “2” and “3” in order to perform a task. Meanwhile, node “C” might need to obtain locks on resources “1” and “3” in order to perform a task. Theoretically, if all three nodes attempt to obtain locks on these resources at roughly the same time, then node “A” might end up with a lock on resource “1,” while node “B” might end up with a lock on resource “2,” while node “C” might end up with a lock on resource “3.” Under such circumstances, all three nodes would be waiting for another of those nodes to release its lock on a resource. Node “A” would be waiting for node “B” to release a lock on resource “2” before node “A” could release the lock on resource “1.” Node “B” would be waiting for node “C” to release a lock on resource “3” before node “B” could release the lock on resource “2.” Node “C” would be waiting for node “A” to release a lock on resource “1” before node “C” could release the lock on resource “3.” As a result, a deadlock condition occurs, in which all three nodes end up waiting indefinitely unless some outside intervention occurs.
Sometimes, these deadlock conditions can be detected automatically. For example, if a monitoring process determines that a database server instance has been waiting for an excessively long period of time, then the monitoring process may conclude (perhaps incorrectly, though) that the database server instance is involved in a deadlock condition. In order to break the deadlock, the monitoring process may take some action. For example, the monitoring process might instruct the apparently deadlocked database server instance to terminate itself by exiting gracefully. In response to such an instruction, the assumed-to-be-deadlocked server instance releases all of the data resource locks that it currently holds, and ceases execution. As a result, other database server instances that were involved in the same deadlock condition can obtain the terminated instance's released locks and resume their work.
The foregoing approach works sometimes. Other times, however, the reason why a particular database server instance is taking an excessively long time to complete its processing is actually due to some reason other than a bona fide deadlock condition. Sometimes, for example, the hardware configuration of a particular storage subsystem that the particular database server instance needs to access causes that access to be unusually slow, thereby causing the database server instance to take much longer than is normal to perform its work. A monitoring process, noticing the excessive amount of time that the database server instance has taken thus far in pursuit of its current task, might erroneously conclude that the database server instance is either deadlocked or “hung.” In order to allow other database server instances to obtain locks on the resources on which the delayed server instance currently holds lock, the monitoring process may terminate the delayed server instance in the manner discussed above. However, under the situation described in the foregoing example, terminating the delayed server instance might actually exacerbate rather than solve the problem.
In the situation described above, after the lock-holding database server instance has been terminated, another server instance that was waiting to obtain a lock on the same resource will attempt to obtain that lock and access that same resource. However, if the reason why the terminated process was taking so long to complete its task relative to the resource was due to the fact that the resource is stored on a very slow piece of hardware, or on a piece of hardware that is being heavily accessed at the moment, then this other database server instance will probably experience the same delay that the terminated server instance did. The monitoring process might end up taking the same action—termination—relative to the other database server instance also. If many different database server instances are all seeking to access the same resource (possibly because the resource is or contains critical or popular data), then the ultimate outcome might involve the termination of most, if not all, of the database server instances in the cluster. In short, the whole cluster might be brought down when it ought not to have been.
Current approaches for dealing with these kinds of conflicts in a cluster suffer from shortcomings such as the type discussed above largely because current approaches for resolving conflicts treat each conflict detection and resolution as a separate, unrelated, independent incident. However, in reality, conflicts sometimes are related to each other. Sometimes, conflicts are self-repeating. Sometimes, multiple conflicts are all related to some temporal event. Because current approaches for detecting and resolving conflicts do not account for these possibilities, current approaches often take a significant amount of time, and do not actually determine or resolve the root cause of the problem (which may continue to repeat itself). Current approaches for conflict detection and resolution are somewhat defective in that they do not consider a high-level, overall view of the system, and in that they do not perform any kind of analysis to determine whether multiple problem incidents are related to each other.